Skip to content

REQ: keep the sketch consistent when an update fails - #808

Open
leerho wants to merge 1 commit into
mainfrom
req-update-failure-safety
Open

leerho wants to merge 1 commit into
mainfrom
req-update-failure-safety

Conversation

@leerho

@leerho leerho commented Nov 19, 2026

Copy link
Copy Markdown
Member

Follows #805 (classic quantiles) and #806 (KLL): the same kind of failure, now in ReqSketch.

Problem

In ReqSketch.compress(), ReqCompactor.compact() trimmed the compacted items from its own buffer and returned the promoted half. The caller then merged that half into the next level, which may have to grow its buffer. If that allocation failed, the promoted items were gone, and retItems and maxNomSize no longer matched the compactors.

update() also set min and max before appending the item, so a failed append left a min or max for an item that was never added.

Changes

  • ReqCompactor.compact() now takes the next level's buffer. It reserves space there first, merges the promoted items in, and only then trims its own buffer and advances its state and coin.
  • ensureEnoughSections() allocates before it changes the section fields.
  • If anything interrupts compress(), the catch block recomputes retItems and maxNomSize from the compactors and rethrows. The sketch stays valid and compresses again on the next update.
  • update() stores min and max after the append.
  • FloatBuffer.ensureSpace() is now package-private.
  • ReqSketch class javadoc: a new paragraph, "Exceptions during update".

A merge can still leave part of the other sketch applied if it throws. This PR leaves that as it is, as in #806.

Testing

  • New ReqUpdateFailureTest:
    • compactorUnchangedWhenPromotionFails: the merge into the next level fails (its buffer isn't sorted), and the compactor's image, coin and the next buffer stay unchanged. Covers both HRA and LRA.
    • sketchConsistentWhenCompressionFails: a ReqDebug throws while a level is being added mid-compression. After that, N, min, retained count, nominal size and iterator weight still agree, and the sketch keeps working and round-trips. This one guards the catch path. It also passes on main, because no hook can inject an OutOfMemoryError at the allocation itself.
  • A temporary probe compared main and this branch over k ∈ {4, 6, 12, 50}, HRA and LRA, n up to 300,000, with and without a merge, using a fixed seed. All 112 cases matched: the serialized images (SHA-256), total buffer capacity, retained count and nominal size.
  • Full suite: 2,437 tests, 0 failures. javac -Xlint:all -Xdoclint:all,-missing reports nothing in the changed files.

🤖 Generated with Claude Code

- compress() no longer loses items when growing the next level's buffer
  fails: ReqCompactor.compact() now allocates that space first, then
  merges the promoted items in, and only then trims its own buffer.
- ensureEnoughSections() allocates before it changes the section fields.
- If compression is interrupted, the retained-item count and nominal size
  are recomputed from the compactors.
- update() stores min and max only after the item has been appended.
- ReqSketch javadoc says what an exception during update leaves behind.

Results, serialized images and buffer capacities are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@leerho
leerho requested review from proost and tisonkun and a balanced review from Copilot November 19, 2026 17:12

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

A throwing compression hook can leave cached queries stale after an update, and iterator allocation failures during recovery can leave compactor counters inconsistent.

2 open issues
What changed in this PR

This PR extends update-failure handling to REQ sketches, following the classic quantiles and KLL changes. It aims to preserve sketch consistency when allocation or compression fails.

Changes:

  • Reserves promotion capacity before moving items and delays section-field changes until allocation succeeds.
  • Updates min/max after appending, repairs counters after interrupted compression, and documents failure behavior.
  • Adds failure-path tests for compaction and continued sketch use.
File Description
src/​test/​java/​org/​apache/​datasketches/​req/​ReqUpdateFailureTest.java Adds compaction-failure and sketch-consistency tests.
src/​main/​java/​org/​apache/​datasketches/​req/​ReqSketch.java Reorders update bookkeeping, adds compression recovery, and documents exceptions.
src/​main/​java/​org/​apache/​datasketches/​req/​ReqCompactor.java Reserves promotion storage and allocates before changing section fields.
src/​main/​java/​org/​apache/​datasketches/​req/​FloatBuffer.java Makes ensureSpace() package-private for compactor use.

Review effort: Balanced

if (Float.isNaN(item)) { return; }
ensureCapacity(); //first use allocates, before anything changes
final FloatBuffer fbuf = compactors.get(0).getBuffer();
fbuf.append(item); //may grow the buffer; nothing else has changed yet
Comment on lines +585 to +586
retItems = computeTotalRetainedItems();
maxNomSize = computeMaxNomSize();

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants